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ABSTRACT : • 

In the first part of this cost-analysis of National 
Science Foundation (NSF) institutes, th^ financing of the' University 
of Wyoming Portal School Project is described and expenditures and 
outcomes bf the program for the years 1971-1974 are estimated. The 
average cost of the project to the NSF was $1.3t) per teacher 
participation *hour. The cost of this-project was compared to that of. 
four other NSF programs. In the second part of this report, the 
cost-effectiveness of various NSF Summer institutes for teachers was 
determined based on teacher and student gains in science test scores. 
The average* cost of a summer institute for physics teachecs J^as 
$58.50 per teacher percentage point gain in test score. The ' 
cost-effectiveness of another summer institute in physics is given on 
the basis of increase an student perf orm'Snce. on three tests. (BB) 
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^ ' Introduction . ' 

. ; Costs are sbadowy figures hdver/Lng in thfe backg^und 

of-,evaluation, spectres that (jome to haunt those »who Vl 
. tried to ignore them, Janus-faced figures more elusive 
^^than Che^mdst ghostly of the mental entities to which 

the' hard-nosed empiricist, objects in the scientific ^ 
^. context, and yet the very sub^^tance of the hard-nosed 
empirici^t^s position in the management area. 
XSqriven, 1973) 

' '^la this raither eloquent sentence, Scriven has emphasized two crucial 
points concerning cost-analysis in eiducational evaluation. First, program 

'Costs are' in Important aspect of evalyation. The ,evaluatqr ^ho ignores 
proM^m costs i^ omitting an important component of prtgram evaluation. 
Th.e decision-mtke^^ who neglects to consider program costs 'is* basing his 
decisions on only "t)artial .i-nforlnation and, consequently, risks the . ^ ^ / 

possibility of prrc^rs of ^^dgment-Vv^ ^ ■ , 

] ' ' ^ ^ \ ^ ' ' *' ' 

The second poi^t ^^criven 'makes concerns thfe >extreme ,dif fdculty. of both 
^ ' ' " . ^ i ' ' . *^ V ■ 

^- ^ * * ' * ' " , ' 

obtaining cost data and defining hosts'. Costs are generally classified as 

4 f - 

direct or indirect. IXirect costs include such things as salaries, tuition,; ^ 

and materials costs. These are not aS difficult to 'define as to obtain.'^ " 

Indirect costs are a bit mote elusive. They arp. difficult to both define* 

and' obtain. Overhead and preparation time' are indirect costs... If -la teaclie^\ 

decides not to take a summer job in order^ to attend a workshop,, this is an*' * ' 

Indirect co§t. If . an institute attendee devfflLops an ulcer 'attributabl^ ta • 



parti<;ipation in- the program, this, too, is an indirect cost. In view of 
thq[ problems with-meaeuring indirect costs, this study is concerne(5 mainly 

.^with direct ^costs. , • / * • ♦ 

. The present study is a tvd-pait exploration in cost-analysis of ^ 

\ National'€"d^enc^' Foundation (^^ prdgrams. .The two parts' 

• rep^tpsent different;' aspects of "6f fecAveness. " Part I is concerned ^th' 
.^oarticip^iid^^h^antJ; exposure costs. /"Participation cost" foi: teachers is tfie 
. case*af pi!pyidih&/6iie^ tfeicher with the opportunity to, attend an NSF 

r sponSoreil'jE^^graili.^ .St^^ "exposure cost" is the cost for one qtudenu to 
•receivfe the 'benfef>t-^of- a teacher's attendance.' ^ • i 

^ Th^^*secbnd part of the sttidy is Concerned with determining the cost 
of v^riou% outcome measures* for teachers ' partiicipating in NSF sponsored, 
'prcigraias ai^*fpr students who a;ne affected by these teachers. These 
,jnea^ur^s are'4>^rh4ps.;inor^ crug4.al to 'a cost-analysis tfian participation and 
^xposur^ co^ts.;^, It 'is one problem to determine the cost of^.,pr0viding a 
.•teacher with^ the oppor4>Jdnity to attend a workshop or an institute. It is 
"an altogetljer different issue to determine if these opportunities are 
jsroducing a posit^-^ ofiange 1^" the" teachers and the . teachers ' students, 
a9d if so., what these 'gWLn^ are^costing. Various^ NSF sponsored summer 
institutes will be examined In terms of^ these gains. • / ^ , 

• ; These e^mples by no Wans- exhaust ef f ectiveli^ess measures*. A com- 
preherfsive^' long-jranged cost* analysis would take into account such, things 

as occupational .choice pf program •participants,, increased earnings, benefit ^ 
td tjie national Interest, ,etc. - But these measures are beyond* bot^i the sc/pe 
.and interest. of this study. 



.PART I: PARTICIPATION AND EXPOSURE* COSTS 

The Portal Schopl Concept ^ . » J * ^ 

The Portal Schoof Projec^ of the University of Wyoming was chosen as a " 

case -study for this cost-analv4is for two reasons. First, the* project is 

an innorvative example of NSF Sponsored p?:gg^ams. ^edoncHy^ estimates of 

both cost and effectiveness measures were available.' • , , . 

\ The 'Portal Schopl Pi:pject was a part of one of f iye ^r9grams to * . 

receive^ 'funding" under the NSF Comprehensive Grants l^ogram in 1971. For 

those interested in a detailed portrayal of the Portal" Schools, see . ■ 
I ... • . 

'Evaluation of the Portal* School Program (Bracht et* al., 1973). ^The 

purpose of the Portai School program is to assist schools* in tjie ^Wyoming 

^reg^^on *^o 'dfevelop their capacity f-er self -improvement in science and 

mathematics education. The program is operated by the Science Ad 

Mathematics^ ^Teaching Center (SMTC) at -^he University of .Wyoming. The ^ 

Center is staffed 'by the Collage of Arts and Sciences and the College of' / 

' % \ \ ^ 

"Education. The SMTC' is the focal- point of both science and mathematics' 

educational research and n:eaeher training. 

^ r^^^ steps exist in the formulation and implementation of a portal 
Schoo^. ^The first step involves, assessing the needs of and formulating 
objectives for participating school disttJ.cts. Representatives frtnn the 
district meet with the coordinator of the Portai' School program* If the 
decision is made to implement a Porta.1 School, the coordiliator also assists 
ip.th^^lection of »the Portal leadei:(syi • , • 

The second phase- concerns <he training of thesre J.eaders. Generally,.. 
prospf^c^iAte leaders are brough ^to the 'University of Wyoming campus du^ring 
the summer to atteftd a minitute. A minitute follows a generally unstrtifTtured 



environment. Prospective Portal leaders, being aware of the specific needs 
and general Portal School plan of the district, are in formed ^-ibout available 
materials, facilities' and faculty, arid prermitted to explore alternative 
programs* ^e miititute lasts anywhere from one to six weeks". ^ THe end 
result is a detailed' worlo.plan for the particular Pcjrtal School. 

After the Workshop plans^'have be^n approved' and the leader (s) ' ^ 
certified, the lead^(s) r.ettim' to the school district to conduct the 
'Portal School. Portal School workshops are conducted ^t <Jne of three 
levels;^ exploration, workshop participants may examine. one or iflore\ curricula 
implementation workshop's prepare teachers to implement a specific curriculum 
and; creative exp^sion workshops allow teachers to modify ^d ekpand upon 
curricula. 

Each wor)cshop 'carries one, two or three semester credits from the 
University ctepending upon whether the iPSO minute sessions arefeid seven, 
eleven or seventeen times, respectively. The^duration of the workshops is. 
determined by the needs of the pa;:ticular Portal School, 

^ Financing thfe Portal School 

Introduction ^ ■ • , * ♦ - ^ 

In the- evaluatipn report (Bracht et. al., 1973), sources, of Pdrtal 

School support and types of ^expenditures were described in detail. Figure 1 
* « • • » 

presented. below, .is a modified versioti of Figur^ 1 (p, 45) taken from that 
report^^ The present study 'xd.ll expand this effort by ^providing cost ' 
figures fofr a^'^hree year period, 1971. to 1974. The sources of support ^(i.e. 
financial resources) represented in Figure 1 are described below.. 
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Figure 1 



Matrix of Expenditures and Financial Resources 
in the University of Wyoming's Partal School 



Expenditures 

* « 

A. Pield Coordination 



B. Development of University 
Services ^ 

C. Training Portal Leaders 



D. • Salary ;for Portal Leaders 



if Curriculum Materials 
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NSF ^Contributions^ 

• • • * 

NSF funds are used mainly to train Portal lea'ders, Oth^r uses are* , 

the salary and travel expanses of thif coorditiato^r. The' latter is used^tn 

settlor up Portal Scliools. N:ost figure^ wfere provided by the SMTC. The 

figiires wer^ taken from the financial reports after completion of the 

^ project. , 

NSF funds were granted t9 support science works'hops. No money wenf • 
directly for math education. - However, -a percentage of the tuition paid bW 

. science participants revels back to the SMTC (see "Local' Contributions" ^ 
oanextpagfe). Some of this money" i-s- used' tfd^initiate'workshops' for • • " ^ 
mathematics teachers. In e3sence, th^n, '^SF money iSy|^ the foundation for 

'both sQience ^nd^^thematips workshops'.' Both science^d math participation 

* ' \ V : . ' ' ' 

hours (definelfl- below) are therefore included In the a^at^sis. / 



University Expanses 



/ 



The Services of the SMTC are essential in the operation of the Portal 



School Program. 



The SMTC provi)ies staff, facilities and materials for. the 



training of ^^ortal leaders* Occasionally the Center- lends materials for use 

in* workshops. The figures. for both university expenses and l^al contribii- 

tions .are estimates provided by the SMTC of fin^|[icial contributions" to t>^* ' 

entire comprehensive program. 'Most of the money, however., went tto th& 

< ' ^ * * ^1 

Portal »School Program. University^ figures include released .time ^dr faculty 

v> " ^ ' 

and staff as well as small sums for materials.. Because* only a small portion 
was lised for materials, the' entire sum was distributed proportionally** in the 
calculation between .science and math partici^pation hours.' The feeling was*' 
^expressed, ^however, that slightly more money was spent on math materials 
• The lat£cr had to be constri/cted while science materials j^ere generally 
packaged and ob^ined economically from the distributors.' . 



Local Contrll^tlons ^ * ' . . 

The » operations of 'the Portal, Schgols ate ^gr^atly dependent upon local 

' ' 9 , 

district\support, I'or example, the unlversitjr charges a tuition fee at 
the rate of $20 per semester credit for any Portal School participant.. The 
.SMTC, estimates that over half of the tuitioh fees are paid^by the school 
districts. The other ^lalfi is paid by the teacher^.* Tuition received by 
the 5MTC is distributed 'as follows:' J s ' • ' 

30% stipends for Portal leaders 

30% held in reserve for school systems tp purchase 

•additional PortaL School services • ^ • , 

40% revolving fund of the SMTC ' ' - . 

Tuition is noj|:' the only expense incurred by the school districts. They 

must provide facilities for workshops as well as inateifials, texts, etc. • 

fistimates of school dis'trict expenses were provided jjy the SMTC. These * 

figures -were obtained from a sample of letters estimating expenses f rpm 

participating^^lclTools. These estima^tes 'are direct costs^nd "include ''such 

« •> 
things as travel 'for participants, kits, texts and other materials. 

/' , . ' 

Science teachers pa^^ticipatlug in Portal School workshops, ;Ln lS>71-72 

• ' • * 

and 1972-73 were sampled to obtain estimates* of expenses that they them- • ^ 

Selve^^were required to pay. The most coimnonly listed expense was tuition." * 

Estimates of tuition are included above. Other expenses included minimal 

/ > • . ' ' ' 

trav^, 'texts, etc.- Much of this was reimbursed by, the district. In 

essence then, reported, non-reimbursed teacher expenses are negligible and 

are not included in this^ report. ' , \ 

^. ■ •. . . • '\ 

- s Uata Collection ' * * . * ^ 

Cost figures f or« the sources of financial contributions described ab^ve 
are presented ^n Table 1. As mentioned above, sources and estimates of these-> 



figures were obtained from -the staff of'the-SMtC, Explanation of some of* 
•the -other figures listed in the table is_ necessary at this poin't. 

Figures for workshop participants are also, listed in Table K The 
figures presented have been rounded, Becaus^^of rounding etror, the 
ambi,tious reader who decides to recalculate some of the measures may derive 
slightly different figures. ;A. teacher , semester refers to one teacher 
participating in one\vorkshop regardless, of the number ^f credits received 
and .regardless of the number of other workshops this hyp(3theticai teacher 
^attended. Participation hours ref ers"~to^length, in hours, of the workshops 
\ (i.e., length .of • the session x number of sessions), Minitute attendees'are 

hot included in "the estimates. ' ' » 

♦ • - ' ' ' . 

Workshop participant data -data were* compiled '"from various sources. Some 

of the data was gathered personally by the author. Additionil data were' 
• s^nt by t^ SMTC staff at the request of^ the author..' Still other data Were 
'Obtained in conjunction with another study being conducted at the' M^esota' 
'Research and Bvaluatfon Proje,c» (MREP). .in addition, a partial list of . 
•voi*kshop -participation figures been compiled by the "SMTC ^taff for their 

own purposes. , < - 

. Because the qniversity Records were noSti, kept in anticipation of^|;his 
type of study, complete accuracy of "figures is difficult to obtain. Where 
duplicate data were available from both the SMIC ^n'd the author, cross- 
checks were made. Various 'datV categories for 1971-73 ''(S.g. *1971-72 
elementary math) were d is crej^anf fr6m zero to approximately ten ^rcanT. 
The only large discrepancy appeared betwen the duplicate figures for'l973-" 
74. The-SMTC estimated 1,564 total teacher semesters • (sub-category data ' 
estimates were not available^, while the author estimatea 1,765 teacher ^ 
semesters. These A&Vsl were gathered as an afterthought, to the original two 



k * ■ • Table 1 ^ . . 

Estimates of Expenditur,es and Outcomes, for the Pprtal School Program 

Year of Program. 
1971-72.' ^ 1972-73 1,973-74 - 
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Number o-f Science 
Teacher Semesters 

Number^£<Math 

Teacher "Semesters • 

Average Parti^cipation 
^ Hoiifs - Scie^ice ^ 

Average Participation 
Hours - Math 



N§F Funds 

r^slimated University 
^ Eunds 

Estimated Local ^ 
Contributions 



Total Oosts 

Cost /Teacher Partici- 
pati9ri Hour y^SF 

Cost/Teacher ''Partici- 
pation Hour - Total 

Usage Rate of Curricula 
(initial and .subsequent 
years)/ ■ ' 

1972- 73 

1973- 74 

1974- 75 ^ 

Percelitage Who Uged 
Prpgjam at -Least 
Oncfe'^by 1974-75 

NSF Cost/Student 
(initial and subsequent 
years) ^ 3-971-72 

. ^ 1972-73 

1973-74 

' ' . . 1974-75 



985 
584 

28 \ 
$ 52,022'* 

'-$ 61,3i3 

$ 82^00 
$195,>635 



.X 719 
1187. 



'40*- ' 



.27 . 
.$ 88,030 

$ 71,339 
$137,900 



$ 



. $297,269 

.9a \$ 1.5b 



3.60 



46% 
54%. 
46% 
43% 

64% 



, ^8% 
47% 
. 38% 

58% 



$1.5j0 
$ .70 
$; .50 

$ ..4b 



'$2.30 • 
r- 

$1.10 . 
$ .80^ 



Estimate 



It* 



-Data •Not Av^ilabfe for 1973-74 
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C 939 
826 
27 
"26« 
$ 76V-960 

$119,275 
$194, 980- 



3-Year 
Totals/ 
.Averages 

• 2643 ' 

2517.- 

• ' 35 ; 

* k 

27 • 
012 



$251,927' 

$415,180 
$884, 



$391,215 •• 

V 

$. 1.60 $ . 1.30 
4^90 i $ 8.20 > $ 5.40 



. ** 



^ • , : -.10 

year study,, and circvunstances. did not permit' as careful consideration of this 
year as- the ^wo previous years. The discrepancy rate in total teacher ^ 

.semesters for 1971-74 between the figures provided by the SMTC ^nd those 

.con5>iled by this author is less t|;ian thr^e percent. 

' : . : . " 4 • 

Figures for teacher participation are brokea down by year and by ' 

* \ 'Hi- 

liscipline. The fiscal year used is June 1 to May 31; that is, in, order ' 
to b>-ihcluded in the 'fiscal year, a workshop had to be virtually completed 
by May 31.^ ^This seemed "to^be the most natural break in the year according ' 
to the maimer in which workshops- were -scheduled and avoided arbitrary I 
decisions concerning which fiscal year' to place various workshops. 

.... . • ^ ' 

^ Outco me Measures ' ' ^ I i 

' Outcome taeas^i^s are also- listed in Table 1. The unit used to asses0 
dosts is teacher participation hour. That is, ho v much does it cost foV ohe 
teacher to participate in an NSF sponsored program for one hour? This * 
measure was s^jelecte'd aT>er number , of teachers because the latter proved 
misleading. For Example, one math teacher might'^^'have.p'articipatecl'^ljci a one 
credit worksljop while one' ^science teacher participated in a thre'e credit 
workshop^ It hardly seems f^ir to estimate 17.5 hour^s of instruction as 
costing the same as %2. 5 /hours. * , ' - 

A seconda^jj^measure^ is; cost per stu|den't exposed to ^a trained teacher. I ' 
The number -of- students/affected can- be viewed in x)ne of two ways. First, 
on^an take ^he7pp,sitibn that any^teacher attending a Portal School sub- \ 
sequently will have some effect -on all students^ (s)he encounter's. ^ ' \ 
Alternatively, It c^ be argued, that students benefit only from* direct • 
applica-tion of part or 'all of a program studied at the PortaUSchool. The 
cost per student figures provided in Talple 1 are based on the latter ' 
assumption. . * • ^ , ' ^ ^ ' 
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Cost per studerft^estimates along with usage. rate (explained below) were 
obtained from a questionnaire sen t^^ to 4 ^random sample of teachers who 
.attended, science workshops during 1971-7 a.. . These data were coUected as part 
of anorCHerv study being conducted at ME?. Data are not-, available, on mathe- 
matics workshops norw^n science workshops conducted during* 1973-74. . 

, Sampling of attendees was stratified according to yeaiNattended, type of 
cui^cula (i.e., elementary or secondary) and type 'of region Durban vs. 

. rural). The latter distinction was liot retained in the calculations; the 

* ■ * « 

population had been stratified in thi,^anner for other purposes. Calcula-"' 
t tions performed bgth with and without the urban-rural stratification 

revealed no significant; differences and,, in most cases, no differences at all 
Minltiite attendees, respondents who had slipped into our sample by mistake, 
(i.e.4^math workshop attendees along with people who had never ifttended a. 
Portal School at all) and respondents who returned blank questionnaires for 
^ne. reason or another, were elliainated from the sample. For the remaining 
respondents, estimates of number of students affected and program usage rate 
are based on attendees' 'reports of whetfter or not a program was studied at^ 
the Portal School and subsequently implemented 'iir the classroom. Any 
program studied at tihe Portal School and ^plelnented at any level of usage 
(i.e. , supplementary, high or full) was included in- |:he estimates. Where it^ 

• was indicated that a teacher used a program but neglected, to report the 
number of students, the m^n* number of students for user% of that strata was 
substituted." • ' 

■ -r. ■ 

The cost per student estimates take into account that teachers offen . 
attended more than one workshof) in a given year. Also taken into account is 
the fact tfbat .teachers occasionally attended workshops in successive years. 
Therefore, in calculating the number 'Of students affected due to attendance 
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at 1972-73l^rkshops, any teacher who attended a i^orkshop in 1971-72 was 
eliminated fromvthe sample as his/her students were alre^y Included In the 
1971-72' estimates* Essentially, then,. 1071-72 was thought of In this paper / 
as the starting point of the program,-^ In realito* the program began In 1970. 

^^t per student Is provided for each successive yea* beginning with 
'the, year -the workshop, was held. The cost per student decreases each year ^ 
because the .initial (NSF) Portal Schopl cost J remain fixed, but teachers ■ ^ 
affect additional students' each year. 

Usage rate figures are also displayed in Table 1. Usage rate r^ers to 
what percentage of attendees (regardless of the numb^gf workshops attended) 
^used part or all of a^Portal School program that year. Usage rates are 
provided for each successlv^year beginning .with the>ear in which the work- 
shop was scheduled. In\addition, a cumulative usage rate, the percentage 
of workshop attendee's who had ever used part or all of any program they 
- studied at a Portal School at least onc# by 1*974-75, is provided. 

Some cautionary r^rks must be Imparted concerning the figures for both 
usage rate and "cost per student. The figures represent this author *s best 
^ estimate .of program usage and exposure costs. However, these estimates were ' 
obtained from self-reports of attendees concerning whether or^ not.they\had 
studied a particular curriculum,^ subsequently implemented the ^curriculum to 
any degree, and how many of their students were "exppsed',' to the curriculum. 
Anyone who has worked wftfi this type of data is e^^are of, the fallibility of . 
the human memory. For example, giany respondents were uncei^g^n of the dates 
they studied a particular curriculum while ofhers, according to our records, 
reported' dates ^incorrectly. Where more than one respondent yas available 
.from the^^ame workshop, cross-fehfecks were 5iade concerning what the participants 
Studied and when they studied it. 

; — : ■ ' '15 • • . 
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A second problem" concerns the largd number of ,non-respindents (48%) 
from our sample* Because a non-respondent's ^tudy was not conducted, it 
becomes a bit hazardous to generalize beyond our sample. 



Conclusion 



It is difficult to make any conclusive statements about the coSt outcomes 
provided in Table 1.. What does it mean to 6a^ that it costs NSF $1.30 for a 
teacher to participate in an NSF .sponsored program for one hour? Pe^rhaps the 
question that should be asked ^s "Is it worth $1.30 per hour to' train a 
teacher?" Alternatively, "Is it worth this money to 'reach the teachers' 
Students?" A partial answer to this question,' perhaps, lies in whether or 
not this expenditure is helping to achieve NSE goals. What is $1.30 per . 
hour bijjfing? Are, teachers and students receiving benefit from these programs?* 
This issue will be .discussed in the second part of this paper. 

Another possible answer to this problem is ^perhaps contained in a compar- 
ison of various delivery systems. A comparisoi^of this type is provided below. 

' ' ■ ' ' - 

Comparisoti of Deliyery Systems * 

. Cost effectiveness measures provide a meaningful basis for certain ^ 
kinds of decisions. Cost comparisqns can provide additional useful information. 
Table 2 (Welch- and Willson,^ 1975) provides estimates- of costs and effectiveness 

if 

measures for f ive ifi^fferent delivery systems. Estimaftes for 1973-74 are 
provided for the four additional delivery systems. Estimates foj: 1972-73 
are included for the Portal School simply because mor^inf ormation was • 
available f or ^hat year. The reader who feels" uncomfortable with this ' 
decision is urged to refer back to. Table 1 to make additional comparisons.. 



u 



Criterion 



Table 2 ' 

Delivery 'System Comparison ^ '^^^^^^ 



: J 

eliv^ry 



Deliv^y System 

A^gessible ^ Off -^Campus : Collab- Portal Summer 

Schools Center^ orative Schoot Workshops 

(Mississippi) ' (Notre Same) (S.Dakota) (Wyoming) (S. Dakota) 



Number of 
Teachers > 



234 



75 



92 



a906 



61 



^ Estimated 
NSF Cost 



' $100,000 



$115,126 $27,34^^ $88,030 .$15,527 



Hours of 
Instruction 



45 



90 



38 



32 



80 



NSF Cost Per " \ . ^ 
Teacher Partici- * 
pation Hours^-^ $9.50 



Usage Kate " 84% 




$1.50 



48% 



$3.20 
51R 



Cost Per Pupil 
(adjusted) by 
. Usage Rate 



$6.20 



$35.90 



$14.10 



$2.30 



$6.70 



This figure repres^V^s teacher semesters. The data indicate that approxi- 
mately 13t of science teachers attend^ed two workshops "during the year. 



For science workshops only. 



■ \ 



Reported by participai(ts jluring first-year after' trd^jdjig; recent 
research suggest^ these figures attenuate considerably^(Ln subsequent 
years. ' 
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^ PART II: ^ALTERNATIVE EFFECTIVENESS MEASURES ^ , 

Introduction ^ * . ° " 

■ - ' ' JL i 

*A goal of NSF i^; to iiKrease stieritific literacy. Some of their 
efforts in this direction have taken the f orm ,of institutes for 'teachers. . 
.It is of interest, to determine the cost of participation in these institutes 

but thi^ is not the complete story. It is also important to determine if 

•V 

these insti-t^tes are, helping .to accomplish the goals of NSF; it is important 
to determine if these "institutes are having any differential effect on both 
the teachers attending and the students that these- teachers aff^^t. 

Various NSF summer Institutes w^re selected for study because data are 
available, both on program costs and effectiveness measures. Flrslj'^ ' 
institutes will be examined In terms of the effects on teachers partlcl- 
patlng In the Institute and ^ the corresponding costs. The second 4)art will ^ 
ekomln^ the effects on the secondary audience, "^t he students, and the costs 
of resulting changes In various outcome m^asu3;es. 

^ Teacher Outcomeg- ..i.r " " 

* ^ . ' *■*. , •> * 

Outcome measures for teachers were , obtained during the course of "An 
Evaluation Sf Summer Institute Programs for Physics Teachers" (Welch and 
Walberg, 1968). Four summer Institutes were randomly selected for study*. 
Institutes were selected on* the basis of the following criteria:' 1) they 
were six weeks In- duration, 2) they began on or 'kfter June 27, 1966, 3) 
they were-deslgned for a single suminer of .study, and 4) they were designed . 
specifically for physic^ teachers. , * * ' • ^ . 

The directors of these fqur Inst^utes agreed to adnjinlster a pr^-post 
battery of Instruments to.t;h^ .teachers attending /the institutes. The 
Instruments were admini^steVed on t^ first and last days of ^ the institute ^ 

. ■ 18 ■ ,. 



The instruments are describedv below: ^ 

• * v 

TOUS Test 'on Understanding Science. Designed to assess 

understanding of the scientific enterprise; scientists, 
* and the aims of science ♦ (Cooley and KLopfer, 1961 )♦ 
^^^r . (Form Science Process Inventory, j^eveloped to 

assess kn<jwledge of the activities, assan5)tfons, ethics, 
and producta of science. (Welch and Pella, 1967-68). 
TST? .Test on Selected Topics in Physics. Tests knowledge of 
t>hysics but emphasized hist^orical, philosophical and 
interdisciplinary items. iSelected Harvard Project 
Physics Test J-tems). 
One hundred<-and $i:fty-two persons began the program. ^ Testing took place 
over two days and, not all workshop attendees were present for all tests. We 
do know"Cti£t at ieast 15A persons completed- the courses. Where ^t was 

•V. ' ) ' ' . . / ' ^ " 

evident that ^person^ om a particular institute missed, a particular test 

^ v-|i . ^ ' ^ 

tjae mean for that institute 'was substituted for all statistics listed in 

'^^^^^4"'* '.This seQq|e ^re'approptiate/than substituting the grand me^ 

because! th^e wei^f ♦significant differences iji^pgains amons institutes. Eight 

per'sons still remain unaccounted for. It is dot •known by this researcher if 

these eight 4>eople ever completed the cdurse* 

Tabled provides a cost-analysis for the suoii;er institutes, .figures' 

presented in^the table have been rounded and the ifeader is 'again reminded , 

that any recalculations based on these figures must take roun4ing error into ^ 

consideration-. The effectiveness measure used is cost per percentage point - 

-^in. Percentage pojtnt gain essentially means how jnany points the mean score 

was raised on ^ instrtwent compared with (i.e. , ^divided by) howjaafiy^bints 

the mean could haVe increased iip to a perfect score. The ire^der will note 

that^three sets o^ cost figures ^re provided. This is for the benefit of ^ 
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.those^whp feel-'tliat tha^results should be generalized to 162 people or, 
for -that matter, should remain confined to those who took 'the tests. . ; 

Total NSF costs fo^r the four in'stitutes were estimated at $166,130. 
It must be kept, in [mind when cpnsidering cost outcomes that these workshops 
were conducted several years ,ago. NSF financial policy decisions (e.g., 
reduction of stipends) along with 'economic conditions of the time^.g., • • 
inflation) need to be taken into ''account wheu considering cost outcomes ofj 
any particular year. . ' \i 

• . It.--is--abvious-^hairifcTi¥se~ teachers ^ demonstrating significant gains on 
► tliese measures. The problem to be pondered is whether it is worth $166,130 
to raise 154 teachers three points on a test -of physics achievement. Put 
differently, is it worth $58.50 to raise one teacheij one percentage point on - 
a test? Perhaps a partial, answer to- this questi'DiT lies in whether or not 
students are recefiving benefit from their teachers'* attendance 'pf' these 
institutes. ' - . • • - - 



Student Outfcomes , 



It would seem from the previous analysis that^'these institutes have a 
positive effect on its attendees, ^e next logical step is to explore the 
effects, if any, that are transmitted to the students and the corresponding 
costs. * ^ • " ' S 

The program selectedTbr study was the Harvard Project Physics Summer 
ijistitute held at 'Wellesly in. 1967. Outcomes measures are available from the 
evaluation report, A National Experiment in Curriculum Evaluation (Welch and 
Walberg. 1972). Cost d^^a was provide'd by the Harvard Project Physics staff 
and includes travel expenses, stipends, instructional costs , and room, board, 
and incidental services provided by T^ellesly. * Total expenses were $58/837. 
Again, NSF policy changes and economic conditions must be t^ken into considera- 
tlon when computing present or future cost outcjcnaes. 
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Table 

Corst-Analysis of Teacher Outcomes 



• Instrument . " ^ TOUS 

Number Test Items 60 ^ 

IJximber "^amiilees 147 

Pre-test 'Mean (N«154) ^44*07 

PosV-test Mean (1^154) ^ 46.02 

. . Possible Gain (N»154) ^ . .15.93 

Actual 'Gain (N=154)" ^ * . ^ 1.96 

■ Ayerage Percentage Gain (N=154) , 12.3% 

- Cost/Percentage Point Galh^(N«154) $87.80 

" Cost/Percentage Point Gain (N=il62) $83 .'40 

' Cost /Percentage;^© iYit Gain ' $88.30 
* 5l " No. of Examinees) 



TSTP . 

40 
153' 
23.33 
26.40 
16.67 
3.07 
18.4% 
$58.50 
$55.60 
$59.60 



Figures have been rounded 
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During the summer, a national random sample of 34 teachers was Tjrougfit 

to Wellesly for- a six-week training session on Project Physics^ a physics 

course for high school students. The following year these teachers taugjjt 

thq course. < variety of. instruments were administered in * a" p?e-post ^ 

format td^ these teachers' classes \N«1583) and to.a^vcontrol-griup (N»853). 

Three of these measures yerre chosen fpr analysis: Te^t^Q Understandii^ 

Science (iCOUS) , Sciericei^rocess Inventory (SPI) , ^d Phyafics Acliievement 

Test (PAT). The latter is' a local\y developed test of 'general topics in 

phy§ics' (Winter & Welch, 1967). , . ^ ^ ^ 

i ' ^ ^ ^ 

Table 4 provides a cost -analysis 6^' student, outcomes with regard to the 

three instruments. The* analysis is viewed in terms of cost per pe^ntage 

point difference; that is, how^much does it cost tc^ raise one student *s 

test score one pfercentage point above a control grpup's percentage point 

i / • , ' ' • ' ■ 

-galji. ^ ^ . ^ ' 

• /. <• 

AgadUi, it would seem 'that these infetitu^es are having a positive 

• .... - ' 

effect dn the secondary audience, the^ students of the teachers attending 

an institute. Those students' show greater achiev^ent tfhan the control 

group. In some, cases, thi§ gain is small (SPI); in others (PAT, TOUS), 

the gain is greater. The ^ type of question that still' temains i^- whether 
» * » 

it Is worth $58,837 to raise 1583 students an average of f^A .petce^tage >h 
points on a Test Of Uitderstanding Science over *«and above students whose 

tkndi 

.Conclusion ^ 



teachers had not attended institutes. ^ ' ^ , ^ * 



Obviously, I have a^^^ne a great deal of side-stepp'ing with regard to 
making judgments of cost outcomes* In partial answer to whether it Is 
cost-effective for teachers to participate in NSF sponsored lnstt^te§, ' 



Table 4 

'Co&t-Analysis of Student Gairfs , 



Instrument V 

Number Test Items . /♦ 

Pre-test Mean (Experimental) 

Post-test Mean ^Experimental) 

Possible Gain (Experimental) 

Actual^Gain (Experimental) 

Percentage Points Gained 
(Experimental) 

i 

Pre-test Mean (Control') 
^ Post-test Mean (Control) 
^Possible Gain (Control) 
Actual Gain (Control) 

"> 

Percentage Points Gained 
^.^'^ ^ ^(Control) 

^Percentage Points Dlifefenc/e 
Cost/Peircentage Point Difference 



r 



TOUS 


PAT 


■ 60 , • 


40 


V 

32.7 


18,4 






35.5 


24.3 


£,7 i 3 


21.6 






2.8 


5^9 


<L0.3% 


27.3% 


33.5 


-19.2 


. ' 34.5 


23.1' 




on Q 


1.0 ' 


3.9 


3,8% 


. 18.8% 


• 


8.5% 


$5.70 


$4.40 


\ 
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I have lacked toward gains on various outcome measures. In partial answer 

to whet;her these teacher gains are ^ost-ef f ective , I have raised the point 

as to whetheV students are receiving Benefit. But given that both the 

) • 

teachers and the students ^re gaining on the measures 'discussed above,, it 

- ' ■ - \ . / « , ^ ^ r 

still remains to be detehnined whether or not (the gains are worth .the feosts. 

It-is not my purpose to' make these judgments • - I havi -neither the 
inclination nor qualifications to do so. The real pt^pose of this paper ^ 
is twofold^ First, I..hope L have provided exploratory information that 
might be useful in future decision-making. Secondly, I hope I have 
stimulifed interest ^in an aspect of. evaluation that needs both further 
attention and exploration. Traditionally, evaluators hive not been 
trained in cost-analysis-. Neither are reqords kept in anticipation of 
such analyses. In 1969, Robert Stake had" this to say abou^cost-analysis 
in educational evaluation: "It embarrasises me to admit th3t I know notl^ng 
about the measurement of costs," ^ I hope this^ is changing and I have reason* 
to believe it is. Recently, I hav^ seen both pleas and models for cost- 
analyses appearing In ^the. evaluation literature, 

lawould especially like to thank the staff of th>s<SMTC and Harvard 
Project Physics', along with the 'Comprehensive Project Directors for thQir , 
efforts and patieace in assisting with this study providing information 
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